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EXECUTIVE  SUMMARY 


During  the  week  of  21  July  2014,  members  of  the  Joint  U.S.  Forces  Korea  Portal 
and  Integrated  Threat  Recognition  (JUPITR)  Advanced  Technology  Demonstration  (ATD) 
supported  a  U.S.  Air  Force  (USAF)  team  in  the  evaluation  of  the  BioFire  FilmArray  system 
during  the  USAF  exercise,  Beverly  Midnight  14-03,  at  Osan  Air  Base,  South  Korea.  The  goal  of 
the  USAF  team  was  to  conduct  an  initial  evaluation  to  see  if  the  once-fielded  FilmArray  system 
could  eventually  replace  the  Joint  Biological  Agent  Identification  and  Diagnostic  System 
(JBAIDS)  for  environmental  sample  analysis. 

JUPITR  ATD  and  USAF  members  analyzed  environmental  samples  using  the 
FilmArray  system  and  JBAIDS  in  accordance  with  the  USAF  Biological  Detection  Concept  of 
Employment.  Exercise  participants  specifically  focused  on  implementing  potential  materiel, 
doctrinal,  and  procedural  changes  to  reduce  the  time  and  costs  associated  with  analyzing  samples 
from  aerosol  detectors  and  collectors,  such  as  the  Dry  Filter  Unit  (DFU). 

The  FilmArray  system  showed  enormous  potential  to  rapidly  screen  samples  for  a 
wide  range  of  biological  targets.  However,  known  issues  with  the  instrument’s  hardware, 
software  (version  1.4),  and  commercially  available  BioThreat  assay  pouch  caused  USAF 
members  to  question  the  instrument’s  configuration  as  a  potential  replacement  for  the  JBAIDS. 

In  light  of  these  observations  and  lessons  learned  during  the  exercise,  the  JUPITR  ATD  members 
made  four  recommendations. 

•  Recommendation  1:  Given  that  sensitive  detection  of  pooled  DFU  samples 
will  be  critical  for  the  economical  implementation  of  biological  field  testing,  members  of  the 
Joint  Project  Manager  (JPM)  Guardian’s  Common  Analytical  Laboratory  System  (CALS) 
program  should  perform  a  side-by-side  assessment  of  the  FilmArray  system  and  JBAIDS  to 
define  the  polymerase  chain  reaction  (PCR)  performance  envelope  for  the  FilmArray  system. 

•  Recommendation  2:  To  ensure  the  armed  services  fully  adopt  the  FilmArray 
system  for  environmental  sample  analysis,  members  of  the  JPM  Medical  Countermeasure 
System’s  (JPM-MCS)  Next  Generation  Diagnostic  System  (NGDS)  program  must  derive 
sensitivity  requirements  for  any  government-developed  PCR  assays  from  current  JBAIDS 
performance  metrics. 

•  Recommendation  3:  The  JUPITR  ATD  members  should  repeat  the  Beverly 
Midnight  exercises  using  hardware  and  software  upgrades  suggested  by  BioFire  personnel, 

(1)  to  evaluate  the  effectiveness  of  these  vendor-provided  upgrades,  and  (2)  to  ensure  that  USAF 
members  are  fully  satisfied  with  the  FilmArray  system  as  a  potential  replacement  for  the 
JBAIDS  for  environmental  sample  analysis. 


iii 


•  Recommendation  4:  Given  the  inconsistent  performance  exhibited  by  the 
commercial  BioThreat  assay  pouches,  the  NGDS  program  members  should  institute  the  BioFire- 
recommended  instrument  software  and  hardware  upgrades,  assuming  that  the  upgrades  are 
proven  effective.  They  should  also  consider  initiating  possible  quality-control  assay-development 
measures  with  BioFire  members  to  ensure  that  the  FilmArray  system  and  its  assays  are  consistent 
and  reliable. 
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The  work  described  in  this  report  was  authorized  under  contract  number 
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advertisement. 
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ASSESSMENT  ON  THE  CAPABILITY  OF  THE  NGDS  PROTOTYPE  TO  REPLACE 
THE  JBAIDS  FOR  ENVIRONMENTAL  SAMPLE  ANALYSIS 


1.  INTRODUCTION  AND  BACKGROUND 

Beverly  Midnight  is  a  set  of  annual  installation-wide  exercises  designed  to  test 
and  evaluate  the  emergency,  recovery,  and  defense  operations  of  the  5 1st  Fighter  Wing  (the 
resident  tenant  unit  at  Osan  Air  Base,  South  Korea)  in  simulated  disaster  or  emergency 
environments.  During  the  exercises,  members  of  the  Joint  U.S.  Forces  Korea  Portal  and 
Integrated  Threat  Recognition  (JUPITR) 

Advanced  Technology  Demonstration  (ATD) 
supported  the  evaluation  of  the  U.S.  Air  Force 
(USAF)  Biological  Detection  Concept  of 
Employment  (BDCoE).  The  BDCoE  provides 
USAF  installations  with  a  standardized  set  of 
service-wide  employment  concepts  that  govern 
the  use  of  biosurveillance  assets.  These  concepts 
are  used  to  provide  installations  with  a  layered 
approach  to  defend  against  biological  warfare 
attacks,  with  a  specific  focus  on  the  efficient 
implementation  of  environmental  detectors  and 
collectors,  such  as  Dry  Filter  Units  (DFUs).  The 
JUPITR  ATD  members  supported  the  evaluation 
of  the  BDCoE  by  analyzing  spiked  DFU  samples 
using  the  Joint  Biological  Agent  Identification  and  Diagnostic  System  (JBAIDS)  and  the  BioFire 
(bioMerieux,  Inc.,  Durham,  NC)  FilmArray  system. 

The  JBAIDS  is  currently  fielded  to  units  across  the  Department  of  Defense 
(DoD).  It  is  a  polymerase  chain  reaction  (PCR)-based  system  that  is  used  to  identify  a  range  of 
biological  targets  in  clinical  and  environmental  samples.  The  ruggedized  JBAIDS  includes  a 
sample  carousel  that  enables  users  to  choose  the  system’s  multiplex  sampling  capabilities  (i.e., 
[1]  screening  one  sample  for  up  to  five  targets  using  no  controls  or  [2]  testing  four  or  more 
samples  for  a  single  target).  The  JBAIDS  requires  a  minimum  of  60  min  for  sample  preparation, 
which  includes  40  min  of  instrument  run  time. 

The  Next  Generation  Diagnostic  System  (NGDS)  Increment  1  platfonn  will 
eventually  replace  all  JBAIDSs  in  the  field.  The  Joint  Project  Manager  (JPM)  Medical 
Countermeasure  Systems  (MCS)  members  selected  the  BioFire  FilmArray  as  the  commercial 
system  for  the  NGDS  Inc  1  program  (pending  an  ongoing  and  upcoming  protest).  The  FilmArray 
system  is  a  PCR-based  instrument  that  can  analyze  a  single  sample  during  a  60  min  run  using 
highly  multiplexed  assay  pouches. 

BioFire  Diagnostics  (Salt  Lake  City,  UT)  manufactures  a  range  of  pouches  that 
include  target  assays  that  are  tailored  to  a  specific  application  or  mission.  Each  pouch  requires 
about  10  min  for  sample  preparation.  The  BioFire  BioThreat  assay  pouch  is  used  to  analyze  a 


Figure  1.  Participants  shown  loading 
samples  onto  the  BioFire  FilmArray 
system  during  the  USAF  exercise, 
Beverly  Midnight  14-03. 
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single  sample  for  the  presence  (or  absence)  of  1 7  biological  warfare  pathogens  (2 1  targets  in 
total). 


Figure  2.  The  JBAIDS  (left)  enables  operators  to  choose  the  system’s  multiplex  capabilities 
but  requires  60+  min  for  sample  preparation,  whereas  the  Film  Array  system  (right)  uses 
highly  multiplexed  assay  pouches  that  require  <10  min  for  sample  preparation. 


2.  ANALYTICAL  APPROACH 

During  the  Beverly  Midnight  exercises,  members  of  the  USAF  51st  Medical 
Group  and  JUPITR  ATD  evaluated  the  BDCoE  by  analyzing  spiked  samples  that  originated 
from  DFUs  located  across  Osan  Air  Base.  There  were  two  main  goals  for  the  Beverly  Midnight 
exercises: 


•  compare  the  BioFire  FilmArray  system’s  environmental  analysis  capabilities 
to  those  of  the  JBAIDS  during  a  blind  evaluation  of  spiked  DFU  filters,  and 

•  evaluate  the  FilmArray  system’s  biosurveillance  capabilities  in  accordance 
with  the  BDCoE. 

During  the  exercises,  bioenvironmental  engineering  specialists  from  the  51st 
Fighter  Wing  collected  DFU  filters  every  12  h  from  sites  across  Osan  Air  Base.  As  seen  in  the 
Table,  some  DFU  filters  were  reconstituted  with  10  mL  of  spiked  phosphate-buffered  saline 
solution  containing  one  or  more  of  the  following  inactivated  or  killed  biological  agents: 

•  Bacillus  anthracis  (BA)  Ames, 

•  Yersinia  pestis  (YP)  Kim5,  or 

•  Venezuelan  equine  encephalitis  (VEE)  Trinidad  donkey  1 A/B. 
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The  spiking  concentrations  were  based  on  results  from  the  U.S.  Army  Edgewood 
Chemical  Biological  Center  (ECBC)  BioSensors  Team  Test  Bed  that  was  conducted  in  2012: 

•  5  x  103  cfu/mL  for  BA, 

•  5  x  103  cfu  /mL  for  YP, 

•  1  x  106  PFU/mL  for  VEE,  and 

•  blank  (unspiked)  samples  used  as  controls  throughout  the  exercises. 


Table.  DFU  Samples  Prepared  for  Analysis 


FilmArray 
System  Pool 
Number 

Number  of 
Pooled  DFUs 

Concentrated  with 
InnovaPrep? 

Spiked  Analyte(s)* 

PI 

5 

Y 

BA 

P2 

3 

Y 

YP,  VEE 

P3 

5 

Y 

BA,  YP,  VEE 

P4 

5 

n 

BA,  YP,  VEE 

P5 

3 

n 

[blank] 

P6 

5 

n 

BA,  VEE 

P7 

5 

n 

YP 

P8 

3 

n 

BA,  YP,  VEE 

P9 

5 

n 

BA 

P10 

10 

n 

BA,  YP,  VEE 

Pll 

3 

n 

[blank] 

P12 

10 

n 

[blank] 

P13 

3 

n 

BA,  YP,  VEE 

*  Spiking  concentrations  =  5  x  103  cfu/mL  for  BA,  5  x  103  cfu  /mL  for  YP,  and 
1  x  106  PFU/mL  for  VEE. 


As  seen  in  the  Table,  various  DFU  pooling  schemes  were  evaluated  using  a 
pre-established  pattern  of  3,  5,  or  10  fdters.  From  each  spiked  or  unspiked  reconstituted  DFU 
fdter,  3  mL  of  sample  liquid  was  used  for  pooling.  Some  pooled  samples  included  multiple  target 
analytes.  In  addition,  some  pooled  samples  were  concentrated  using  the  InnovaPrep  (InnovaPrep, 
LLC,  Drexel,  MO)  concentrating  pipette  to  determine  if  the  device  increased  the  likelihood  of 
detecting  low-level  concentrations  of  target  analytes. 

The  pooled  samples  seen  in  the  Table  were  analyzed  using  two  instrumentation 

suites: 


•  the  5 1st  Medical  Group  members  analyzed  pooled  samples  using  the 
JBAIDS  and  associated  DoD-validated  assays,  and 

•  the  JUPITR  ATD  members  analyzed  samples  using  the  BioFire  FilmArray 
system  and  commercial  BioThreat  assay  pouch. 
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At  the  direction  of  USAF  leadership,  the  5 1st  Medical  Group  did  not  use  the 
JBAIDS  to  analyze  samples  for  VEE,  although  some  pooled  and  spiked  samples  contained  this 
target  analyte. 


3.  OBSERVATIONS  AND  CONCLUSIONS 

During  the  Beverly  Midnight  exercises,  the  JUPITR  ATD  and  USAF  participants 
recognized  the  BioFire  FilmArray  system’s  capabilities  to  potentially  replace  the  JBAIDS  for 
environmental  sample  analysis.  The  USAF  leadership  was  specifically  intrigued  by  the 
capabilities  of  the  instrument’s  multiplexed  assay  pouches,  which  were  especially  effective  when 
evaluating  samples  that  had  little  to  no  intelligence  of  the  potential  biothreat.  USAF  members 
also  viewed  the  assay  pouch  as  a  potential  force  multiplier  because  of  its  simplified  workflow 
and  integrated  sample  preparation  capabilities,  which  significantly  decreased  the  training  burden 
associated  with  the  instrument. 

However,  the  JBAIDS  outperformed  the  FilmArray  system  in  terms  of  assay 
sensitivity.  It  should  be  noted  that  the  JBAIDS  assays  are  not  multiplexed,  whereas  those  of  the 
FilmArray  system  are  highly  multiplexed.  Multiplexing  often  results  in  slightly  lower 
sensitivities  due  to  competition  for  reagents.  The  difference  in  multiplexing  likely  had  an  impact 
on  the  analysis  of  the  samples  that  were  spiked  with  multiple  target  analytes  during  the  exercises. 

Specific  exercise  sampling  included  the  following  results: 

•  the  FilmArray  system  correctly  identified  only  five  of  the  13  pooled  samples 
seen  in  the  Table  (a  38.5%  true  positive  rate), 

•  one  false  positive  occurred  on  the  JBAIDS  (sample  PI  1), 

•  zero  false  positives  occurred  on  the  FilmArray  system, 

•  five  false  negatives  occurred  on  the  FilmArray  system,  and 

•  an  additional  four  cases  occurred  in  which  the  FilmArray  system  reported 
positives  only  for  the  Bacillus  species  and  not  the  more  specific  BA. 

The  exercise  participants  also  experienced  software  errors  on  the  FilmArray 
system  that  were  not  reported  until  after  the  runs  were  completed  (approximately  65-75  min 
after  sample  receipt).  The  software  showed  that  an  assay  was  invalid,  and  not  enough 
information  was  available  for  the  user  to  ensure  the  success  of  future  runs.  JUPITR  ATD  and 
USAF  staff  recognized  that  these  errors  were  directly  linked  to  a  previously  identified  issue  in 
which  the  USB  and  FireWire  cables  that  connected  the  instrument  to  the  laptop  became  loose 
and  resulted  in  a  software  error. 

The  JUPITR  ATD  staff  also  experienced  issues  with  some  BioThreat  commercial 
assay  pouches,  which  failed  to  draw  hydration  solution.  This  issue  was  seen  in  half  of  the 
pouches  from  a  single  box  (i.e.,  three  out  of  six  pouches).  Because  of  these  issues,  no  pouches 
from  this  box  were  used  for  sample  analysis. 
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There  were  multiple  instances  during  the  exercises  when  the  BioFire  BioThreat 
assay  pouch  meta-analysis  identified  the  presence  of  one  or  more  BA  targets  in  the  sample  but 
issued  a  summary  call  of  positive  for  the  Bacillus  species.  The  instrument’s  software  algorithm 
did  not  indicate  the  presence  of  BA  unless  all  three  assay  targets  were  present.  The  JUPITR  ATD 
members  acknowledged  that  BioFire  users  must  be  on  the  alert  for  the  false  identification  of 
environmental  samples  as  containing  BA  when,  in  fact,  the  samples  contain  other 
environmentally  ubiquitous  near  neighbors. 


4.  RECOMMENDATIONS 

The  participants  in  the  Beverly  Midnight  exercises  noticed  a  distinct  difference  in 
the  sensitivity  of  the  FilmArray  system  compared  with  that  of  the  JBAIDS  (the  FilmArray 
system  results  showed  a  positive  success  rate  of  38.5%  during  the  exercises,  which  was 
significantly  lower  than  that  of  the  JBAIDS,  although  both  instruments  were  used  to  analyze  the 
same  samples).  This  was  likely  because  multiple  target  analytes  were  spiked  into  samples,  which 
forced  each  individual  assay  on  the  FilmArray ’s  commercial  BioThreat  assay  pouch  to  compete 
for  reagents.  The  ECBC  members  noticed  similar  results  for  the  BioThreat  pouch  during  the 
BioSensors  Test  Bed  evaluation  in  which  assay  sensitivity  for  samples  containing  only  YP  C092 
was  an  order  of  magnitude  lower  (better)  than  samples  that  contained  BA,  Vaccinia,  and  VEE 
(5.0  x  10°  versus  5.0  x  101  cfu/mL). 

The  JUPITR  ATD  members  recognized  the  labor  savings  associated  with  the 
multiplexed  BioThreat  pouch  and  understood  that  a  trade-off  existed  in  tenns  of  assay  sensitivity 
for  samples  that  contained  more  than  one  biological  target  (which  could  occur  when  pooling 
samples  from  DFUs  located  across  an  installation).  Therefore,  the  JUPITR  ATD  members 
recommended  that  additional  analyses  be  performed  on  the  FilmArray  system  to  detennine  the 
commercial  BioThreat  assay  pouch  performance  for  DFU  samples  that  contained  single  and 
multiple  target  analytes.  These  perfonnance  results  were  compared  to  those  for  the  JBAIDS. 

Because  the  JPM  Guardian  (JPM-G)  Common  Analytical  Laboratory  System 
(CALS)  supports  the  environmental  sample  analysis  requirements  for  the  NGDS  Inc  1  program, 
the  JUPITR  ATD  members  made  four  recommendations  to  ensure  that  the  CALS  and  JUPITR 
ATD  assessments  would  be  perfonned  in  parallel. 

Recommendation  1;  Given  that  sensitive  detection  of  pooled  DFU  samples  will 
be  critical  for  the  economical  implementation  of  biological  field  testing,  the  JPM-G  CALS 
program  should  perform  a  side-by-side  assessment  of  the  FilmArray  system  and  JBAIDS  to 
define  the  PCR  performance  envelope  for  the  FilmArray  system. 

In  addition,  the  JUPITR  ATD  members  recognized  that  the  JPM-MCS  team  could 
develop  a  government  off-the-shelf  assay  pouch  for  use  on  the  FilmArray  system  to  meet  some 
environmental  sample  analysis  requirements  for  the  CALS  program.  They  recommended  that  the 
JPM-MCS  leverage  these  perfonnance  characteristics  for  any  government-developed  PCR 
assays,  once  the  perfonnance  envelopes  for  the  JBAIDS  and  FilmArray  system  were  fully 
characterized. 
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Recommendation  2:  To  ensure  the  armed  services  fully  adopt  the  FilmArray 
system  for  environmental  sample  analysis,  the  JPM-MCS  NGDS  program  must  derive  sensitivity 
requirements  for  any  government-developed  PCR  assays  from  current  JBAIDS  performance 
metrics. 


Exercise  participants  also  experienced  hardware  and  software  errors  on  the 
FilmArray  system.  These  errors  were  not  reported  until  after  a  run  was  completed,  which  was 
approximately  65-75  min  after  sample  receipt  (i.e.,  a  previously  identified  issue  with  connecting 
the  FireWire  data  cable  caused  a  known  software  error).  A  JUPITR  ATD  member  spoke  with 
BioFire  Diagnostics  representatives  about  the  FireWire  issue.  The  BioFire  team  developed  a 
solution  and  incorporated  it  into  the  next  iteration  of  the  instrument  (version  2.0).  The  separate 
FireWire  and  USB  cables  that  were  being  used  were  replaced  by  a  single  Ethernet  patch  cable 
(Cat5e).  The  new  instrument  had  onboard  data  processing  capabilities  and  the  capability  to  store 
data,  if  connection  with  the  laptop  was  interrupted. 

The  exercise  participants  also  experienced  issues  with  the  commercially  available 
BioThreat  pouches.  During  the  exercises,  the  JUPITR  ATD  staff  used  three  pouches  that  did  not 
properly  draw  hydration  fluid  into  the  pouch  when  a  sample  was  introduced.  A  JUPITR  ATD 
member  discussed  this  issue  with  someone  from  BioFire  Diagnostics,  who  sunnised  that  the 
failures  were  due  to  the  sample  loading  stations  intermittently  pinching  the  pouch,  thereby 
disallowing  hydration  of  the  reagents.  BioFire  members  redesigned  the  loading  station  and 
provided  the  JUPITR  ATD  team  with  the  newest  iteration. 

Because  BioFire  implemented  potential  hardware  and  software  solutions,  the 
JUPITR  ATD  members  suggested  that  the  staff  should  repeat  the  Beverly  Midnight  exercises  to 
determine  if  the  upgrades  properly  address  any  issues  experienced  during  the  exercises. 

Recommendation  3:  The  JUPITR  ATD  should  repeat  the  Beverly  Midnight 
exercises  using  the  hardware  and  software  upgrades  suggested  by  the  BioFire  team,  (1)  to 
evaluate  the  effectiveness  of  the  vendor-provided  upgrades,  and  (2)  to  ensure  that  the  USAF  is 
fully  satisfied  with  the  FilmArray  system  as  a  potential  replacement  for  the  JBAIDS  for 
environmental  sample  analysis. 

If  the  BioFire-recommended  software  and  hardware  upgrades  are  found  to 
increase  the  performance  and  consistency  of  the  FilmArray  instrument  and  commercial 
BioThreat  assay  pouch,  the  NGDS  program  should  institute  these  upgrades  for  all  DoD-fielded 
FilmArray  instruments  moving  forward.  The  JUPITR  ATD  members  also  recommend  that  the 
JPM-MCS  members  consider  an  inspection  of  the  BioFire  production  line  for  the  commercial 
BioThreat  pouch  and  the  other  U.S.  Food  and  Drug  Administration  (FDA)-approved  assay 
pouches,  (1)  to  help  identify  any  potential  discrepancies  that  may  exist  between  FDA-  and  non- 
FDA-approved  production  lines,  and  (2)  to  provide  recommendations  from  the  DoD  to  increase 
the  consistency  and  reliability  of  the  BioThreat  pouches. 
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Recommendation  4:  Given  the  inconsistent  performance  exhibited  by  the 
commercial  BioThreat  assay  pouches,  the  NGDS  program  should  institute  the  BioFire- 
recommended  instrument  software  and  hardware  upgrades,  assuming  that  the  upgrades  are 
proven  effective.  They  should  also  consider  initiating  possible  quality-control  assay-development 
measures  with  BioFire  to  ensure  that  the  FilmArray  system  and  its  assays  are  consistent  and 
reliable. 
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ACRONYMS  AND  ABBREVIATIONS 


ATD 

Advanced  Technology  Demonstration 

BA 

Bacillus  anthracis 

BDCoE 

Biological  Detection  Concept  of  Employment 

CALS 

Common  Analytical  Laboratory  System 

DFU 

Dry  Filter  Unit 

DoD 

Department  of  Defense 

ECBC 

U.S.  Army  Edgewood  Chemical  Biological  Center 

FDA 

U.S.  Food  and  Drug  Administration 

Inc. 

Increment 

JBAIDS 

Joint  Biological  Agent  Identification  and  Diagnostic  System 

JPM 

Joint  Project  Manager 

JPM-G 

Joint  Project  Manager  Guardian 

JPM-MCS 

Joint  Project  Manager  Medical  Countermeasure  Systems 

JUPITR 

Joint  U.S.  Forces  Korea  Portal  and  Integrated  Threat  Recognition 

NGDS 

Next  Generation  Diagnostic  System 

PCR 

polymerase  chain  reaction 

USAF 

U.S.  Air  Force 

VEE 

Venezuelan  equine  encephalitis 

YP 

Yersinia  pestis 
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